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Abstract. In this paper we consider one parameter generalizations of some non - sym- 
metric divergence measures. Measures are relative information, x 2 — divergence, relative 
J- divergence, relative J ens en- Shannon divergence and relative arithmetic and geomet- 
ric divergence. All the generalizations considered can be written as particular cases of 
Csiszar f-divergence. By conditioning the probability distributions, relationships among 
the relative divergence measures are obtained. 



1. Introduction 



Let 



P = (P1,P2, :;Pn) 



Pi > 0,^2pi = 1 } , n ^ 2. 

i=l 



be the set of all complete finite discrete probability distributions. There are many informa- 
tion and divergence measures given in the literature on information theory and statistics. 
Some of these are symmetric with respect to probability distribution, while others are 
not. In this paper, we work only with non-symmetric measures. Throughout the paper 
it is under stood that the probability distributions P,Q G T n . 

1.1. Non-Symmetric Divergence Measures. The followings are some non-symmetric 
measures of information, the most famous among them being % 2 — divergence and Kullback- 
Leibler relative information. All of the following measures can be written in pairs by 
interchanging P and Q, and Pi and The measures are non-symmetric in the sense that 
the expression changes when this interchange is made. 

• x 1 — Divergence (Pearson |15j ) 

(i) x\p\\Q) = ir { p^ = ±rf-i. 

• Relative Information (Kullback and Leiber [ 1 21 ) 



n 



(2) K(P\\Q) = J2pM^)- 

Hi 



= 1 
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(3) 



Relative Jensen-Shannon Divergence (Sibson [T7j . Sgarro [To] ) 

2ft 



n , 

F(P\\Q) = Vftln - 



.Pi + Qi. 

Relative Arithmetic-Geometric Divergence (Taneja [23]) 

i=l \ ' 

Relative J-Divergence (Dragomir et al. |Hj) 



(5) 



2pi 



n , 

D(P\\Q)=J2(Pt-^ ln [ 

i=i \ 



Vi + Qi 
2qt 



Symmetric versions of above measures are given by 

(6) *(P\\Q) = x 2 (P\\Q)+x 2 (Q\\P), 

(7) J(P\\Q) = K(P\\Q) + K(Q\\P) 

= D(P\\Q) + D(Q\\P), 



I(P\\Q) = ±[F(P\\Q) + F(Q\\P)} 



and 



(9) T(P\\Q) = l[G(P\\Q) + G(P\\Q)}. 

After simplification, we can write 

(10) J(P\\Q) = A[I(P\\Q)+T(Q\\P)] 
and 

(11) D(Q\\P)= 1 -[F(P\\Q) + G(P\\Q)]. 

Dragomir et al. jH] studied the measures (jHJ), referred to subsequently as [22] sym- 
metric chi-square divergence. Measure (J7J) is known as Jeffreys-Kullback-Leiber fTJ ^] 
J-divergence. The measure © is Jensen-Shannon divergence studied by Sibson [T7] and 
Burbea and Rao |21E])- Measure (JOJ) is arithmetic and geometric mean divergence studied 
by Taneja jTHj. More details on some of these measures can be found in Taneja [T5] ITU] 
and in the on line book by Taneja [2*Tj . 

In this paper our aim is to work with one parameter generalizations of the non symmet- 
ric divergence measures given by We call these generalizations, non-symmetric 
divergence measures of type s. Also, we call the measure K(Q\\P) the adjoint of K(P\\Q) 
and vice-versa. The same is with the other measures. 
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2. Non- Symmetric Divergence Measures of Type s 



In this section we introduce one parameter generalizations of the measures given by 
(HJ-©- Generalization of the measures © and (0) is already known in the literature and 
has been studied by many authors. Here we refer to it as relative information of type s. 



Relative Information of Type s 

' K S (P\\Q) = [s(s - I)]' 1 



(12) 



*.{P\\Q) 



K{Q\\P) = t^(f), 
i=i ^ ' 

K{P\\Q) = ±p i \n(f i 



E Pitt 
i=i 



a ^0,1 
s = 
s = 1 



for all s£i 

The measure (I12j) admits the following particular cases: 



(i) <$>^i{P\\Q) = \x 2 {Q\\P)- 

(ii) %(P\\Q) = K(Q\\P). 

(iii) $ 1/2 (P| \Q) = 4 [1 - B{P\ \Q)\ = 4h(P\\Q). 

(iv) ^(P\\Q) = K(P\\Q). 

(v) $ 2 (p||q) = §x 2 (p||q). 

The measures P(P||Q) and h(P\\Q) appearing in part (iii) are given by 

(13) B(P\\Q) = Jpm, 
and 



(14) 



1 n 

h(P\\Q) = 1 - B(P\\Q) = -J2(VPi~ Vft) 



i=i 



respectively. P(P||<5) is known as the Bhattacharyya pQ coefficient and /i(P||Q) as 
Hellinger ^U] discrimination. 

From dI21), we observe that $ 2 (P||Q) = $-i(Q||P) and $i(P||Q) = $ (Q||P)- 

We now present new one parameter generalizations of the measures given by (|5))-|J5|). 

• Unified Relative JS and AG — Divergence of Type s 

We consider the following unified one parameter generalization of measures (J3J) and (jlj) 
simultaneously. 



(is) n s (p\\Q) 



FG s (P\\Q) = [s(s-l)}- 1 

F{P\\Q) = EPih tJ * 



i=i ^ 



i=i 



Pi+qi 



G(P||Q) = E( £ ^)ln(^ 

*> i=l ^ 



, a ^0,1 
a = 
s = 1 
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The adjoint of Q S (P\\Q) written as Q S (Q\\P) is obtained by interchanging P and Q, 
and Pi and qi in the expression JTHJ). The measures Q S (Q\\P) can also be obtained from 
(112)1 by replacing pi by — 



-Qi 



We have the following particular cases of fl s (P\\Q) and Q S (Q\\P): 

n_ 1 (p||Q) = n_ 1 (Q||p) = jA(p||g). 
|g) = p(p||g). 

|p) = F(g||p). 
|g) = c(p||g). 
|p) = G(g||p). 

IQ) = |x 2 (QI|P). 
|P) = |x 2 (P||Q). 

The expression A(P||g) appearing in part (i) is the well known triangular discrimina- 
tion, and is given by 



fii) 



in 



iv 



(a) fi (P| 

(b) Qq{Q\ 

(a) fl x (P| 

(b) QriQl 

(a) fi 2 (P| 

(b) fi 2 (g| 



(16) 



A(p||g) = ^ 



{Pi - Qi) 
Pi + Qi 



• Relative J-Divergence of Type s 



We now propose the following one parameter generalization of the relative J-divergence 
measures given by ©. 



(17) UP\\Q) 



'D S {P\\Q) = (s- l)- 1 1 (Pi - «) {^Y' 1 , 1 

D(P\\Q) = ±(p,-q.)\n 

i=i ' ' 



s = 1 



The adjoint of £g(P||Q) written as ( S (Q\\P) is obtained by interchanging P and g, and 
Pi and in the expression (JT7J). 



These admit the following particular cases: 

(i) Co(p||g) = Co(giiP) = A(p||g). 
(h) (a) Ci(p||g) = p(p||g). 
(b) ci(gi|p) = p(gi|p). 

(iii) (a) C 2 (P||Q) = |X 2 (P||Q)- 

(b) C2(Q\\P) = \x\Q\\P)- 



We observe that the relative information of type s, $> S (P\\Q), contains, in particular, 
the classical measures Bhattacharyya coefficient, % 2 — divergence and Hellingar discrimi- 
nation. The unified relative JS and AG - divergences of type s, Q S (P\\Q) and Q S (Q\\P), 
contains, in particular, triangular discrimination and y 2 — divergence, while the relative 
J-divergences of type s, Q(P\\Q) and ( S (Q\\P), yield, in particular, triangular discrimi- 
nation and x 2 — divergence. 
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In this paper our aim is to relate these generalized measures of type s with one another. 
In order to do so, we make use of the Csiszar f-divergence and its properties. 

3. Csiszar /-Divergence and its Particular Cases 

For a function / : [0, oo) — > K, the f-divergence measure introduced by Csiszar [I] is 
given by 

n 

(is) C7(P||Q) = $>/ 

i=i 

for all P, Q G r„. 

The following result is well known in the literature. 

Theorem 1. (Csiszar [HE]^- If the function f is convex and normalized, i.e., f(l) = 0, 
then Cf(P\\Q) and its adjoint Cf(Q\\P) are both nonnegative and convex in the pair of 
probability distribution (P,Q) G T n x T n . 

The generalized measures given in Section 2 can be written as particular cases of Csiszar 
f-divergence (jl8j) . These particular cases are given by the following examples. 

Example 1. (Relative information of type s). Let us consider 

{[s(s - I)]' 1 [x s - 1 - s(x - 1)] , 8*0,1 
x-l-lnx, s = , 

1— x + xlnx, s = 1 

for allx>0 in (Hf. Then C f (P\\Q) = $ a (P\\Q). 




Example 2. (Relative JS and AG - divergence of type s). Let us consider 

{[8(8-l)r 1 [x(^Y-X-8(^)}, 8*0,1 

(20) ^ s (x) = l i=2-xln(^), s = , 

l^ + (^)ln(^), a = l 

for allx>0 in (Hf. Then C f (P\\Q) = Sl 8 (P\\Q). 



Example 3. (Adjoint of Relative JS and AG - divergence of type s). Let us consider 

' [S(S - I)]' 1 [(^Y ~ 1 - S (^)] , 8*0,1 

(21) v s (x) = {^l + \n(^), s = , 



l—X | x+1 
2 



+ 2±lln(5±l), s = l 



for allx>0 in Then C f (P\\Q) = Q B (Q\\P). 
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Example 4. (Relative J- divergence of type s). Let us consider 

(s - i)-\ x - 1) [(^y- 1 - 1] , 8?i 

{x - 1) In (2±i) , s = 1 

for allx>0 in (Hf. Then C f (P\\Q) = ( S (P\\Q). 



(22) Ux) 



Example 5. (Adjoint of relative J-divergence of type s). Let us consider 



(23) q s {x) 



l-x)ln(^i), s = l' 

/or oWi > in Then C f (P\\Q) = ( S (Q\\P). 

By considering the second order derivative of the functions given by (|19j) - (|23|) with 
respect to x, and applying the Theorem ^ it can easily be checked that the measures 
$ S (P\\Q), Q S (P\\Q), Q S (Q\\P), ( S (P\\Q) and ( S (Q\\P) are nonnegative and convex in 
the pair of probability distributions (P, Q) £ T n x T n respectively, for all s £ R for the 
measures $ S (P\\Q), Cl s (P\\Q) and Q S (Q\\P), and < s < 4 for the measures Cs(P||Q) 
andG(g||P). 

Theorem 2. (Taneja [24 ^. Lei /i,/2 : I C R+ — > K. 6e iu>o normalized functions, i.e., 
/i(l) = /2(f) = and satisfy the assumptions: 

(i) fi and / 2 are it/race differentiate on (r,R); 

(ii) there exists the real constants m, M such that ^ m < M and 

(24) rn^JjM^M, f«(x) > 0, Vx £ (r, P) 
inen we nave iae inequalities: 

(25) m C /2 (P||Q) < C A (P||Q) < M C h (P\\Q). 
Proof. Let us consider the functions n m s (-) and n Ms (-) given by 

(26) Vm(x) = fx(x) - m f 2 {x) 
and 

(27) r ?M (x) = M/ 2 (x)-/ 1 (x), 

respectively, where m and M are as given by (|24|) 

Since f\{x) and / 2 (^) ar e normalized, i.e., /i(l) = ^2(1) = 0, then T] m (-) and t]m{') are 
also normalized, i.e., 7/ m (l) = and ?7m(1) = 0. Also, the functions and f2(x) are 

twice different iable. Then in view of (EH), we have 



(28) i£(s) = f'({x) - m tf(s) = (s) -m)>0 
and 

(29) rf M (x) = M f»(x) - fl{x) = f»{x) (m - |^ ) > 0, 
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for all x G (r, R) . 

In view of (J2EJ) and (J2Hj), we can say that 
(r,R). 

According to Theorem we have 

(30) C Vm (P\\Q) = C fl „ mf2 (P\\Q) = 
and 

(31) c VM (P\\Q) = c Mh - h (P\\Q) = 

Combining (jUJ) and fUJ), we get (J2SJ)- 



the functions r/ m (-) and 77m (•) are convex on 

C /l (P||Q)-mC /2 (P|ig)^0, 
MC /2 (P||Q)-C /l (P||g)>0. 

□ 



4. Inequalities Among Generalized Relative Divergences 

In this section we shall relate the relative divergence measures of type s given in Section 
2. The main results of this paper are summarized in the following theorem. 

Theorem 3. Let the generating functions given by iiy\) - ^\) are twice differentiable in 
interval (r, R) with < r ^ R. Then, we have the following inequalities among the 
generalized measures: 

(i) Q S (P\\Q) and$ t (P\\Q): 

^ l^(^) S ~VllQ) 

and 

(33) 7^(^y~ViiQ) 

^P||QK^(^) $ t (P||Q), s + f>l, 

(ii) n a (Q||P) and$,(P||Q): 

1 / r + i\ s - 2 

and 

(35) pfTViia 
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(in) Cs(P\\Q) and$ t (P\\Q): 



r + l\ s 3 / sr + 4 — s 



(36) ( — ) ( ;_ t _ 2 )mp\\q) 



4r*~ 



R+lY 3 fsR + A-s 



< UP\\Q) < {— ) [ 4ip _ 2 ) **(P\\Q), 

0<s<4, i < 2, + 1 

and 

r+l\ s / sr + 4 — s 



< C.(*TO < ^— J ^ 4rf _ 2 J WIG), 

< s < 4, t > 2, s ^ t + 1. 

(iv) UQ\\P) and$ t (P\\Q): 

r + 1 \ s 3 / (4 — s)r + s 



(38) (— r 4r ,w )*»wiq) 



<aQ|| P )<( ^±iV-V(l^±£ 1$l( P 1|Q) , 



2/? y V ARt+2 

< s < 4, i ^ -1, s + t sC 1 

and 

l\ s ~ 3 /(4-s)i2+s 



< 39 > 1-2S- ^-j^i^iw) 



2r / V 4r*+ 2 

< s 4, t > -1, s + t ^ 2. 

(v) Q S (Q||P) andft t (P||Q): 

(40) ^ +1 ( ! ^ 1 ) S * fi *( P H^ 

< n,(Q\\p) ^ R t+1 (^y^J tot(P\\Q), s>-i,t> -i. 

and 

(41) ^(^y WllQ) 

^ ft,(Q||P) < r t+1 ( T -^-) n t (P\\Q), s < -1, t < -1. 



INFORMATION INEQUALITIES 



9 



(vi) <Z a (P\\Q) andQ t (P\\Q): 
(42) r t+1 ( T -^-) (sr + A-s)Q t (P\\Q) 



2 

s-t-l 



^ ( S (P\\Q) ^ R t+1 (^-^) (sR + 4-s)n t (P\\Q), 0^s<4, 



(vii) Cs(QII^) andn t (P||Q): 

r + lV"*" 1 /(4-s)r + s 



(43) ( — ) p — ^)a(P||g) 



fl+lV" M /(4-s)i? + s 



^ Cs(g||P) ^ ^^iT ) { R~ J a(p||g) ' 

< s < 4, i > s, i(4 - s) ^ 6s - s 2 - 4 

an<i 

(44) (-i)"-((i^) ni( p„« 

< s ^ 4, * < s, t(4 - s) ^ 6s - s 2 - 4 

(viii) C^IIQ) andfi t (Q||P): 

(45) (^±±y ' \sr + A-s)n t {Q\\P) 

R+V 6 ; 



<Cs(P\\Q)< [-^) [sR+(4-s)]n t (Q\\P), 

^ s ^ 4, O t, s(t - s + 6) ^ 4(1 + i) 

and 

s-t-l 



(46) ( Sj R + 4- S )Q t (g||P) 



^C-(^IIQ)^ (sr + 4-s)fi t (g||P), 

< s < 4, s < t, s(i - s + 6) ^ 4(1 + t). 

(ix) CWII^P) andn t (Q||P): 

s-t-l 



(47) ^t(^) S [(±-s)R + s\n t (Q\\P) 



< UQ\\P) < (^) [( 4 - + s ] o ^ 5 ^ M > -l. 
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(x) UQ\\P) and( t (P\\Q): 
(48, * (W)" ]mm 



RS+ 1 \ 2 J \ tR + A-t 
Proof, (i) Let us consider 

for all x G (0, oo). 
From ()49jl one has 



24-2 4 x t+l V 2x 



x + lV 2 x(t + l)+t + s-1 1^0, £ < -1, s + t^l 



(50) - - j 4x t + 2 {x+1) 

In view of (150(1 we conclude the followings 



^ 0, t ^ -1, s + t ^ 1 



(51) m= inf g^^x) 



I6MI »— ^(^)", *>-i 

and 



(52) M = sup gy^ix) -- s , , +1 , s - 2 

x6[r,fl] [4^(2^) > 8 + t^l, 

Now dSU and fl52J) together with (j23J) give the inequalities (O and (JSSJ). □ 

The proof of other parts (ii)-(x) follows on similar lines. 

4.1. Particular Cases. Here below we have considered some particular cases of the 
inequalities JH2|)-©- 

• Take t — ~, s = 2 in (|54*(l or in (|36p. one gets 

r< W(P||Q) 2 <g 

• Take £ = ~, s = 2 in (jHSJ) or £ = |, s = 2 in (j3H|) . one gets 



Take £ = 2, s = 2 in fl33J) or in (|3HJ) or in (gDJ) or in (g2I) or in (g7D or in (gHJ) or 
£ = —1, s = 2 in (j54"(l or in (|3fi|) . one gets 
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Take t = 1, s = 2 in or in (pjj). one gets 

' " 2A'(P||Q) 
Take t = 0, s = 2 in or in one gets 

" x 2 WI|P) " 

Take £ = 1, s = 2 in (jHHJ) or in (jSHJ), one gets 



" Vx 2 (Q\\P) " 

Take £ = 0, s = 2 in (jSjj) or in one gets 

r ^ — ^ it. 

V 2K (Q\\ p ) 

Take £ = 0, s = in (gUJ), one gets 

F(P\\Q) 

Take t — 1, s — 1 in (JH3), one gets 

Vg(p\\Q) 

Take t = 2, s = -1 in flHl or t = 2, s = in © or ® or in gTj) or t = - 1 , s 
2 in flUJ) or in one gets 

r < W(P||Q)-{/A(PfTQ) < ^ 



^A(P||Q) 

Take * = -1, s = -1 in <JH2J) or in (JS) t = -1, a = in (J3ED or t = 2, s = -1 in 
(HOD or t = 2, s = in (JI2J) or in (jSj) or t = -1, s = 2 in (jSj) or in (@7D, one gets 



W(qiip)- yA(p||<?) 

Take t = 0, s = 1 in (jHHj) . one gets 

K(Q||P)-2G(P||Q) 
2G(P||Q) 



Take t = 1, s = 1 in (JMJ). one gets 



r < 2G W" p ) <a 

" A-(P||0)-2G(Q||P) ~~ 



Take £ = 1, s = in (J33J), one gets 

1? 



r < v^-V™ < R 
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Take t = 0, s = in (|34|). one gets 

Vm\\p) <R 

Take £ = 1, s = -1 in (gDJ) or in i = 1, s = in (g2j) or (gSJ), one gets 

r< >/a(p||Q) <g 

r ^4 v ^(PHg)- v /A(P||Q) " 
Take * = -1, s = 1 in flU or t = 1, s = in (JUJ) or in (jUJ), one gets 



Take t = 0, s = -1 in (gnj) or t = 0, s = in (g2J) or in (jSJ), one gets 

r ^ *™) 

^ 8F(P||Q)-A(P||Q) " 
Take t = -1, s = in (gOD or t = 0, s = in (HJ) or in (gZJ, one gets 

8P(Q||P)-A(P||Q) 
A(P||Q) 



Take £ = 1, s = 1 in (|46j) , one gets 

6G(Q||P)-P(P||Q) 
^D(P||Q)-2G(g||Pp ' 
Take t = 1, s = 1 in (J4*4"j) , one gets 



£>(Q||P)-2G(P||Q) 
^ 6G(P||Q)-P(Q||P) ^ • 

Take f = -1. -s = 1 in or t = 1. ■<? = 2 in (JUJ), one gets 

^ X 2 (Q\\P)-4G(P\\Q)^ ■ 



Take t = 0. ,s = 1 in (J32jl or t = 1. s = 2 in (|44jl . one gets 

P(P||Q) 

^ p(Q||p)-3P(p||g) ^ 



Take t = -1. ,s = in (|32j) or t = 0. s = 2 in (PHI) , one gets 



Take £ = 0, s = 1 in (|4^|) , one gets 



r< v /4P(P||Q) + 9P(P||g-3 v /P(P||Q) < ^ 
2v^(PW 
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Take t = 0, s = 1 in (|T7jl , one gets 

<JU){Q\\P) +9F(Q\\P) -3,/F(Q\\P) 
Take t — 1, s — in (j4"Uj) , one gets 

r " V /8G(P||Q) + F(Q||F)- v/F(Q||P) " 
Take £ = 0, s = 1 in (gUJ), one gets 



; , ^ v 8G(Q\\P) + HP\\Q) - VWW) < R 
Take t = 0, s = 1 in (jHIJ) , one gets 

€ R. 









" v/2^(Q||P) 


+ G(g||P)- 


>/c(g||p) 


• Take £ = 1 , s = 1 in , one , 


gets 




V2Jf(P||Q) 


+ G(P||Q)- 


^(PHQ) 



^ p. 



Take t = — 1, s = 1 in (021), one gets 



.. , V / 8D(P\\Q) + A(P\\Q)-2 V /A(P\\Q) < fl 
Take £ = — 1, s = 1 in (J44j) or in (J47|) . one gets 



'8B(Q||P) + A(P||Q)-2^A(P||Q) 
Take t = 2, s = 1 in (l4lfl). one gets 



r< 5^iP\\Q)-y/lQD(P\\Q) + X 2 (P\\Q) < R 
" V /16D(P||Q) + X 2 (P ||Q ) _ ^(phq) ^ 

Remark 1. (i) T/ie inequalities hS^) - \4^j admits much more particular cases but we 
have specified here only the one that can be written in simplified form. 
(ii) There are some other similar kind of relations that can't be obtained from the 
inequalities Kffify -fifify such as: between K(P\\Q) and K(Q\\P), between F(P\\Q) 
andG{P\\Q), and between F{Q\\P) andG(Q\\P), between D{P\\Q) andD{Q\\P), 
etc. These can be seen in Taneja |23| 122] and Taneja and Kumar 
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